Integrating machine learning models in multidimensional applications

ABSTRACT

Systems, methods, and other embodiments associated with integrating a machine learning model into a target application are described. In one embodiment, a method includes parsing a definition file that represents the selected ML model to (1) identify one or more input features defined for the ML model, and (2) identify one or more output predictions that the ML model is configured to generate. The input features and output predictions are mapped to locations within the target application. An executable script is generated and configured to function as a data controller or a data communication interface to obtain the input values from the target application and pass data between the target application and the ML model. In response to obtaining the output predictions, the output predictions are inserted in one or more result locations within the target application.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialsubject to copyright protection. The copyright owner has no objection tothe facsimile reproduction of the patent document or the patentdisclosure as it appears in the U.S. Patent and Trademark Office patentfile or records, but otherwise reserves all copyright rights whatsoever.

BACKGROUND

Machine learning (ML) models may be built using various platforms suchas Python, R, SAS, etc. ML models may also be built using variousalgorithms such as regression algorithms, instance-based algorithms,decision tree algorithms and/or other suitable algorithms and models.Due to their complexity, ML models are built and trained by skilled datascientists.

To build an ML model that makes predictions and/or forecasts futureoutcomes, the ML model is trained using data mining of prior historicaldata and complex probability algorithms to predict outcomes. The MLmodel must then be implemented with some application that uses the MLmodel. Typically, the ML model is built in a computing system/platformthat is external to the computing system/platform of the applicationthat will use the ML model. This further adds to the complexity of usingML models by users of the application. In general, users of theapplication are not technical computer scientists and do not have thetechnical knowledge of ML models.

Integration or implementation of an ML model into an application thatuses the ML model requires a custom configuration (e.g., custom coding)based on the type of ML model being deployed and the configuration ofthe application.

Each time a new ML model with a different input or a different output isdeployed, the new ML model again requires additional custom coding toconfigure and connect the ML model with the application.

The custom configuration to implement the ML model requires the skill ofa data scientist with technical knowledge of both the ML model and theapplication. Thus, a non-technical user of the application cannotimplement an ML model without the assistance from the data scientist.Furthermore, the custom coding typically leads to a relatively long andcostly deployment cycle from development to production.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate various systems, methods, andother embodiments of the disclosure. It will be appreciated that theillustrated element boundaries (e.g., boxes, groups of boxes, or othershapes) in the figures represent one embodiment of the boundaries. Insome embodiments one element may be implemented as multiple elements orthat multiple elements may be implemented as one element. In someembodiments, an element shown as an internal component of anotherelement may be implemented as an external component and vice versa.Furthermore, elements may not be drawn to scale.

FIG. 1 illustrates one embodiment of a method associated with training amachine learning model and integrating the model into a targetapplication.

FIG. 2 illustrates one embodiment of a method associated withintegrating/importing an ML model into a target application usingmapping.

FIG. 3 illustrates one embodiment of a model mapping user interface formapping input features and output features between an ML model and atarget application.

FIG. 4 illustrates an example of a visual representation of identifyinginput data locations and result locations from a data source.

FIG. 5 illustrates an embodiment of a computing system configured withthe example systems and/or methods disclosed.

DETAILED DESCRIPTION

The present disclosure describes an integration system and method thatintegrates machine learning (ML) models into a target multidimensionalapplication. In one embodiment, the present integration system allows anon-technical user to import an externally created ML model directlyinto a computing platform and integrate the ML model with a targetapplication without custom coding. The user may then consume/use the MLmodel directly in the target application seamlessly.

In one embodiment, the computing platform may be an enterpriseperformance management (EPM) platform that is configured with thepresent integration system including a user interface(s) that allows forthe integration of one or more available ML models into a targetmultidimensional planning application. The present integration systemand method may be applied to any type of computing platform or targetapplication.

The present integration technique is an improved integration processbased on properties of a selected ML model to automatically map inputfeatures and output predictions of the ML model to appropriatedimensions or intersections in a target application. The present systemallows a non-technical user to easily import and integrate ML modelsinto a target application without knowing technical details of the MLmodel. Once integrated and mapped to the target application, the MLmodel may be used to generate predictions for the target application.

Definitions

The following terms and definitions are used in this disclosure.

“ML model” and “model” as used herein refer to a Machine Learning modeland/or a predictive analytic model. These models refer to mathematicalmodels that use statistical techniques or ML algorithms to learnpatterns hidden in large volumes of historical data. Predictive analyticML models use the knowledge acquired during a training phase to predictthe existence of known patterns in a set of new data. Based on how theML model is configured and trained, the ML model generates and outputsone or more predictions based on one or more defined input features.

“PMML” (Predictive Model Markup Language) is a computer language used torepresent an ML model/predictive model. A PMML file, for example, is amodel file formation that is XML-based, which defines properties of andrepresents an associated machine learning (ML) model. The PMML file maycontain data transformations (pre- and post-processing), datamanipulations, a predictive workflow as well as one or more predictivemodels.

PMML allows a computing system to share predictive analytic modelsbetween different applications without having to generate custom code tore-create the ML model. Therefore, you can train an ML model in onecomputing system or platform, express the ML model in a PMML file, andmove the PMML file to a different target system. The target system caneasily recognize and understand the predictive solution of the ML modelfrom the PMML file. Thus, the target system may then use the ML model(based on its PMML file) to predict whatever values or conditions themodel was trained for (e.g., predicting a likelihood of machine failurebased on a set of data).

“ONNX” (Open Neural Network Exchange) is another language and model fileformat for representing an ML model. An ONNX file may be used in asimilar manner as a PMML file for representing and moving an ML model.

Discussion of various embodiments of the present integration system willbe described with reference to the figures.

With reference to FIG. 1 , in one embodiment, the present integrationsystem may be regarded as having two phases or parts. Namely, a modelcreation phase 100 as shown in FIG. 1 and a modelintegration/consumption phase 200 as shown in FIG. 2 .

During the model creation phase 100, a machine learning (ML) model iscreated in a ML platform and trained for a particular purpose. Forexample, the ML model is trained to make predictions and/or forecastsbased on a set of training data. The ML model may be trained to predictsome outcome such as a likelihood that a computer component will failunder certain conditions, predict if an email is authentic, predictfuture weather conditions, predict future sales of a product, predictstaffing requirements of a company, or any other type of prediction.

At block 110, historical data associated with the desired purpose of theML model is uploaded to an ML platform or retrieved from a database(s).At block 120, the historical data is used to create (train) an ML model.In one embodiment, one or more ML models may be trained in differentplatforms (e.g., R, Python, SAS, etc.) and/or different types of MLmodels can use different machine learning algorithms (e.g., regressionalgorithms, instance-based algorithms, decision tree algorithms, etc.).

The different models may be tested for how accurate their predictionsare based the historical data and the desired purpose. The bestperforming ML model may then be selected as the final ML model. Detailsof training and testing an ML model is beyond the scope of thisdisclosure and will be understood by those of ordinary skill in the art.

At block 130, the final ML model is saved and exported in a model fileformation (referred to herein as “a definition file”) that representsthe ML model. For example, as previously described, the definition filemay be a PMML file (saved as an XML file) or ONNX file. In otherembodiments, the definition file may be created with other model fileformations, for example but not limited to, a scikit-learn model (savedin a pickle file); a Spark MLlib model (saved using the save method ofthe model object); a TensorFlow model (saved in a .pb file); a Kerasmodel (saved in a .h5 file), etc.

Once created, the definition file may then be transmitted and/oruploaded to different computing systems where the ML model is desired tobe used. In another embodiment, whenever new historical data isavailable for the ML model, the ML model may be retrained and exportedas an updated model (e.g., new definition file) that replaces the old MLmodel. The old ML model may then either be archived or discarded.

At block 140, after the trained ML model (definition file) istransmitted to or otherwise imported into a target system, the trainedML model may be integrated into a target application. In one embodiment,integration is performed by mapping ML model input features and ML modeloutput (the prediction results) to appropriate intersections (e.g.,appropriate slices of data) from the target application. The mappingscreate connections and/or associations so that data involved in themappings can flow seamlessly between the target application and the MLmodel. In one embodiment, the target application is a multidimensionalplanning application. Additional details of the integration aredescribed with reference to FIG. 2 .

With reference to FIG. 2 , one embodiment of an integration method 200is illustrated for integrating/importing an ML model into a targetapplication. The integration method 200 is a computer-implementedprocess performed by a computing system that executes executable codeconfigured to perform the method 200. The computing system andintegration method 200 are configured with graphical user interfacescreens that control the steps of integration process and providesselectable options, input fields, and output fields.

In one embodiment, to initiate the integration method 200, a targetapplication is selected and/or a menu option may be provided within thetarget application to import an ML model into the target application.

At block 210, a user interface screen is generated and displayed on adisplay device that provides a list of ML models that are available forselection. The list may include names of each ML model and a descriptionof what the ML model is trained to predict. For example, a model namemay be “VolumeForecastingML” and its description may be “ML model forvolume forcasting based on price, promotion, etc.” This allows a user toeasily identify and select the type of ML model for integration into thetarget application.

In response to an ML model being selected from the list of available MLmodels, the computing system identifies and retrieves a definition filethat represents the selected ML model. Based on the format of thedefinition file such as PMML file, ONNX file, or other type of modelformat file, the system is configured to identify input featuresrequired by the ML model and the output prediction of the ML model fromthe syntax of the file format. For example, each file format has adefined syntax (e.g., specified words, statements, expressions, andstructure) for representing input features and output features of the MLmodel.

For example, a PMML file may have a statement as follows, which isdefines the output of the ML model:

<OutputField name=“Predicted_Volume” optype=“continuous”datatype=“double” feature=“predictedValue”/>. Thus, searching for“outputfield” will identify one of the output features of the ML model.

Based on the file format, the definition file is parsed by the system to(1) identify one or more input features defined for the ML model, and(2) identify one or more output predictions that the ML model isconfigured to generate.

At block 220, the system provides a mapping mechanism to map the one ormore input features of the ML model to one or more input data locationsfrom the target application. The mapping mechanism also provides for themapping of the one or more output predictions of the ML model to one ormore result locations within the target application and/or in a datastorage location associated with the target application. In general, themapping mechanism provides functions to clearly define the datasource(s) of input features for the imported ML model (PMML file) anddefine a target (result location) where “predicted output” generated bythe ML model will be stored or consumed by users as part of the targetapplication.

In one embodiment, the mapping mechanism includes a model mapping userinterface. Once the system identifies the input and output features ofthe ML model from the definition file, the model mapping user interface(UI) is generated based on the ML features and is displayed. FIG. 3shows one embodiment of a model mapping UI 300 to create and configuremappings between the ML model features and appropriatedimensions/intersections/values from the target application.

The dimensions/intersections/values in the target application correspondto the one more input data locations (source locations) from which theML model will receive input values for the input features of the MLmodel. The input values will be used by the ML model to generate aprediction.

With reference to FIG. 3 , the model mapping UI 300 displays the InputFeatures 305 and Output Features 310 that were identified from the MLmodel. All the “Input features” 305 from the ML model need an input inorder to generate a prediction based on the input. An input menu bar 315shows various information about the input features identified from theML model (e.g., feature name and data type). Each input feature mayinclude selectable/modifiable parameters involved in the mapping such as“Display Name” of the input feature and one or more input data locationfields 320 to identify the source location(s) of data from the targetapplication that will be used as input data to the corresponding inputfeature.

In FIG. 3 , example Input Features are shown in a column Feature Namesthat include three inputs: “Product,” “Industry.Volume” and “Price” thatmay be part of an ML model that predicts sales volumes for a product. Ofcourse, any number of input features may be used by an ML model. Anothercolumn identifies a “Data Type” of each input feature. This informationis identified from the PMML definition file of the ML model. Data typemay be any type of data such as string, numeric, double precision, etc.

The one or more input data location fields 320 define the mappingbetween the ML model input features and source data locations from thetarget application. In other words, the source data locations identifythe input data that will be used by each input feature of the ML model.The type and quantify of input data location fields 320 is generallybased on how the data in the target application is organized andstructured. Thus, different types of location identifiers may be used toclearly define the source location(s) of data for each input feature.

In one embodiment, the target application may include data organized asspreadsheets (or forms with data) and may have multiple pages of data.Each page of the spreadsheets may have a unique name identifying thepage and thus the corresponding data on that page. In one embodiment,the model mapping UI 300 contains an option to select a particular pageof data, which is referred to as a cube (“Select Cube Option) as part ofthe input data locations 320.

The selected page is a data source location and constitutes one portionof the mapping that identifies the location of input data that will beused by the ML model. In another embodiment, additional fields may beprovided to identify the scope of the ML model within the selected cubebased on specified types of data within the selected cube, e.g.,account, period, years, etc.

Furthermore, as part of the selected spreadsheet page/cube, the inputdata locations 320 may include fields for identified locations and/ortypes of data within that selected spreadsheet page/cube. This mayinclude “Input type” and “Member.”

In general, a user using the model mapping UI 300 is assumed to haveknowledge of the target application. Thus, the user can map each inputfeature to the appropriate input data parts from the selected cube/pageor other portions of the target application.

In one embodiment, the “Input Type” field is used to define the type ofdata being mapped to the input feature. The input type may be a singledata value (which would correspond to a single location such as aspreadsheet cell value) or a group of data that belongs to a selectedcategory (which would correspond to multiple data locations/cells).

For example, when the target application includes a spreadsheet of data(rows and columns of cells that define multiple dimensions), the inputdata location may be a cell value in a selected page/cube in thespreadsheet (e.g., a dimension member) or a dimension itself (membervalue), which would include all data values that are associated to thatdimension. For example, Input Type=“Cell Value” identifies a specificcell in the cube data structure such as “OFS_Material Unit cost” (e.g.,the name of the cell) or by a column/row cell location (e.g., G12).

In the “Member” column, a “product” value may be a dimension such thatdefines a category of data that is used for input purposes to the MLmodel. Thus, data for all “products” contained in the identifieddimension are used as input to the ML model. For example, “product” mayrefer to all products that have data in a database or may identify aparticular sub-category, which may be a different dimension in the data,like “soft drinks.” The soft drinks dimension may further includesub-categories like cola, diet cola, fruit punch, water, etc. In oneembodiment, the Input Type may be left blank, which will cause the valueto be used as a prompt to receive an input value.

In general, the model mapping UI 300 is configured based on the dataorganization and data structure of the target application to define thedata source for each Input Feature of the ML model. Thus, the possibleinput data source locations that can be selected are changed based onhow the data is organized. For example, if the data is organized inspreadsheets, particular types of source locations identifiers may beused that are associated with spreadsheets. If the source data iscontained in a database or file, then locations are configured toidentify source locations for retrieving the corresponding data valuesfrom the database or file, for example, by query requests or other fileaccess functions.

With further reference to FIG. 3 , in the “Output Features” Section 310,this shows what the ML model outputs as a prediction based on the inputfeatures. In FIG. 3 , the example output feature has the Feature Name“Volume” and has a Data Type of “Double” precision numeric. The outputfeature is then mapped to a result location 325 as part of the targetapplication. The result location 325 is identified in a similar manneras the input data locations 320 in the Input Features Section 305, whichis based on the type of data organization used by the targetapplication.

In the example of FIG. 3 , the result location is shown as “Output Type”and “Member” similar to the input type and member parameters in theInput Features Section 305 since the target application data isorganized in spreadsheets.

Thus, once the ML model generates a result prediction, the resultprediction will be automatically inserted into the mapped resultlocation 325 based on the mappings. This is the target location where“predicted output” will be stored or consumed by users. The resultlocation may be, for example, a one or more cell locations in aspreadsheet, a location(s) on a live display screen, a data record(s) ina database, or other type of location. It is noted that the predictedoutput may contain multiple predications such as predicting a productvolume for each month over a future six month time period.

With reference to FIG. 4 , one embodiment of a spreadsheet 400 (e.g., adata source) is shown that may be associated with a target application.FIG. 4 is used to show an example of a visual representation ofidentifying input data locations 420 and result locations 425, which maybe defined using the model mapping UI 300 from FIG. 3 . It is noted thatthe mappings explained with reference to FIG. 3 do not correspond to theexample in FIG. 4 because they are different mappings and differentexamples. In the example of FIG. 4 , suppose the target application is aproduct planning application and the ML model being imported has beentrained to predict and forecast product sales volumes based on inputsincluding industry volume and price.

The spreadsheet 400 shows data from multiple “Products” 410 in separaterows. Each product row has historical data for “Industry Volume” and“Price” data per month from JAN to DEC from one or more years. The datain these cells are identified as input data locations 420 for the MLmodel. The ML model will use data from these locations as input to themodel to generate a volume prediction for each product for each monthselected. The volume predictions generated by the ML model are selectedto be for each month JAN to DEC in year 2022. The predictions from theML model are mapped to target result locations 425, which correspond tocell locations corresponding to each product row that is part of theinput data 420.

With reference again to FIG. 2 and the integration method 200, at block230, after the mappings of input and output features are complete, anexecutable rule(s) or script is automatically generated based on themappings between the ML model and target application. The mappings aredefined and stored as part of the executable rules/script. In general,the rule/script is configured to function as a data controller and/ordata communication interface to obtain data and pass data between thetarget application and the ML model. Once generated, the rule/script canbe used to execute the ML model and run predictions for the targetapplication. In one embodiment, the rules/script may be generated usinga scripting language, object-oriented programming language, or othercomputer executable language to generate a sequence of statements forperforming the functions of the rule/script.

In one embodiment, the generated rule/script may be embedded orotherwise linked to a form/spreadsheet in the target application. Aselectable option may be provided in the target application to initiateand run the rule (and thus execute the ML model predictions) on-demandwhile the target application is executing and being worked on by a user.The generated rule/script may also be configured as a batch rule/scriptand executed as a batch process while the target application is notexecuting.

In one embodiment, the executable rule/script is configured to definethe data mappings and execute the ML model based on the data mappings.Based on the defined data mappings, the rule/script is generated andconfigured with statements to at least retrieve input data values fromthe one or more input data locations from the target application andpass the data values to the ML model as the one or more input features.

The rule/script is further configured to obtain the one or more outputpredictions from the ML model and automatically insert the one or moreoutput predictions into the one or more result locations within thetarget application. In this manner, the rule/script is configured tofunction as a data controller and data communication interface to obtaindata and pass data between the target application and the ML model. Therule/script may include statements that use one or more applicationprogramming interfaces (APIs) for communicating with the ML model. Therule/script may also include statements that use one or more APIs thatare available on the computing platform to perform the data retrievaland transmission via network components and databases that are part ofthe computing platform of the target application. In one embodiment, therule/script is a groovy rule.

At block 240, in response to the rule/script being initiated on-demandor as a batch process, the rule/script is executed which in turnexecutes the ML model associated with the rule/script. Based on thedefined mappings, data values associated with the mapped input datalocations from the target application are retrieved and passed to the MLmodel as its input features. The ML model is executed and one or moreoutput predictions are generated based on the input features. Therule/script then obtains the one or more output predictions from the MLmodel.

At block 250, the one or more output predictions obtained from the MLmodel are automatically inserted by the rule/script into the resultlocation(s) of the target application as previously defined. This mayinclude displaying the output predictions on a live spreadsheet/form adisplay screen when the rule/script is executed on-demand with thetarget application. When executed as a batch rule, the inserting mayinclude automatically inserting and storing the output predictions intoone or more data records in a database, or other type of data structurefor retrieval and viewing at a subsequent time.

Accordingly, the present system/method provides a technical approach toseamlessly integrate ML models into a target application withoutrequiring technical knowledge of how an ML model is configured or itsrequired data inputs.

On-Demand and Batch Embodiments

In another embodiment of the integration method 200, as previouslystated, the generated rule/script and ML model may be executed on-demandwith the target application or as a batch process while the targetapplication is not executing. Since the rules/script controls theexecution of the ML model, the ML model is executed on-demand or as abatch process in accordance with the rules/script. In anotherembodiment, two separate rules/scripts may be generated and saved, onefor on-demand predictions and one for batch predictions. The executionof the two rules/scripts may be provided as options and controlled byselection via a user interface or menu options.

The following is discussed with reference to the target applicationbeing a multi-dimensional planning application that when executedgenerates one or more data sheets (spreadsheets/forms) with rows andcolumns of data values. Each intersection of the rows and the columnsidentifies a cell location within a data sheet that contains a datavalue. The data values of generated on a live data sheet and displayedon a display device while a user is working with the target application.Conversely, when the target application is not currently executing,there are no live data sheets displayed on a screen. Rather, the datavalues are contained in a data structure (database).

To generate the rule/script as a batch rule, the system configures theexecutable rule/script to retrieve the input data values from the one ormore input data locations by querying the database associated with thetarget application and retrieving the input data values corresponding tothe one or more input data locations from the database. Since the targetapplication is not currently executing, the rule/script cannot retrievethe input data values from a live spreadsheet. Thus, the input data isretrieved from a storage location (database) from where the data ismaintained. Likewise, the output predictions from the ML model areautomatically inserted and stored into the mapped locations in thedatabase. In one embodiment, the input data locations may be identifiedand mapped via a run time prompt.

In one embodiment, the batch rule/script may be configured to accept RTP(Run Time Prompt) inputs for receiving input data (e.g., the dimensionsused for input data) that is mapped to the input features of the MLmodel. Thus, in one embodiment, if a user input is expected during theexecution of the ML model, the mapping UI is configured to prompt foruser input to receive the input data. In such cases when the rule isexecuted, the rule is configured to prompt for user input (e.g., usingRun Time Prompts). In other cases, the input features for the ML modeland output from the ML model are automatically derived based on themappings defined in the model mapping UI. RTP variable limits may be setto the member selection done on the UI.

To generate the rule/script for on-demand execution, the systemconfigures the executable rule/script to retrieve the input data valuesfrom the one or more input data locations that are cell locations withinthe one or more data sheets (see FIG. 3 or 4 , for example). Theexecutable rule/script is also configured to transmit the input datavalues from the cell locations to the one or more input features of theML model.

In one embodiment, the on-demand rule/script may be attached to aform/spreadsheet of the target application and executed through a custommenu or from a rules menu. The rule will run prediction on the membersfrom the form/spreadsheet context. A Prompt' input type from the mappingdone in UI will be prompted for Run Time Prompt RTP input. All requiredinputs are collected from the database, RTP, or member name and sent tothe ML model as input data for prediction. The predicted values aresaved back to the form/spreadsheet grid.

Generating Executable Rule/Script Embodiments

In one embodiment, the executable rule/script in block 230 of FIG. 2 maybe generated to contain the following:

Input definition: Defining a data slice for necessary input features ofthe ML model. When input data for input features are present in datalocations outside of a cube/spreadsheet of the target application, therule/script uses commands to connect to the outside locations using RESTAPIs to connect with external systems such other databases or computingplatforms (e.g., ERP, CRM, etc) to access and retrieve the input datafrom the outside locations.

Preprocessing of input: Any calculations (thresholds, arithmeticoperations, etc.) to be performed on input features before sending theinput data to ML models. This can be used for feature engineering asneeded by the ML model.

ML model definition: Mapping the appropriate ML model i.e., PMML filethat has been already imported into EPM Cloud.

Processing of output: Defining any post-processing steps to be appliedto the ML model output based on the type of predictions being made. Forexample, in a sales opportunity prediction, this step may allow forprediction outputs (e.g., probability of lead conversion) to betransformed into estimated deal size or other form, which can be usedfor other calculations (e.g., revenue estimation).

Output definition: Defining the data slice where the output from the MLmodel should be inserted or pasted.

In another embodiment, the executable rules/script that is generated atblock 230 of FIG. 2 may be configured as follows.

As previously stated, the rules/script functions as a data communicationinterface between the target application and the ML model. Therules/script is generated based on the mappings selected on the modelmapping user interface 300. The rules/script (and thus the ML model) canbe executed from a spreadsheet/form on-demand or run as a batch rule.The rule that runs on a spreadsheet/form assumes the input dimensionmembers are defined on the form whereas the batch rule may takedimension members from a Run Time Prompt (RTP). The rule is configuredto query a database as previously stated for the input values of themapped members and call a prediction API with multiple sets of inputs.Respective output from the ML model prediction will be stored back tothe database against the mapped members for output.

In one embodiment, an algorithm for generating the executable rule mayinclude: getting the mapping information from model mapping UI; createan xml template for the rules; create variables for input from the RunTime Prompt; create variables for all the dimensions from the mappingsexcept any cell value mapped dimensions for batch rule; generate therules/script from the mapping information received from model mappingUI; load the rules/script to a rules processor or execution engine(e.g., a calculation manager); and deploy the rules/script to the targetapplication.

In one embodiment, the rules/script may be generated to containstatements that perform an algorithm where the statements are based onat least the following functions:

Read the form/spreadsheet grid and get a point of view (POV), Row andColumn dimensions and corresponding members. If any dimensions used inthe Cell Value mapping are present in POV/Row, then move them to Column.Remove all other dimensions from Column. Replace the column members withthose from the mappings. Compare all dimension members in the grid withthose selected in the model mapping UI. Use only the common members fromboth. In the case that there are no common members, use the modelmapping members. Create a map of cell intersections excluding mappeddimension members as the key and the corresponding database data as thedata value. Add corresponding member from the intersection to the map asvalue, for input type Member' from model mapping UI 300. Add promptvalues to the map. Pass the values to ML model prediction API. Obtainand store predicted values from the ML model against the correspondingintersection in a map. Form an output grid with the members from outputmodel mapping and save the predicted data to the database of the targetapplication.

Cloud or Enterprise Embodiments

In one embodiment, the integration method/system 100 is a computing/dataprocessing system including an application or collection of distributedapplications for enterprise organizations. The applications andcomputing system may be configured to operate with or be implemented asa cloud-based networking system, a software as a service (SaaS)architecture, or other type of networked computing solution. In oneembodiment the integration system 100 is a centralized server-sideapplication that provides at least the functions disclosed herein andthat is accessed by many users via computing devices/terminalscommunicating with the computing system (functioning as the server) overa computer network.

In one embodiment, one or more of the components described herein areconfigured as program modules stored in a non-transitory computerreadable medium. The program modules are configured with storedinstructions that when executed by at least a processor cause thecomputing device to perform the corresponding function(s) as describedherein.

Computing Device Embodiment

FIG. 5 illustrates an example computing device that is configured and/orprogrammed as a special purpose computing device with one or more of theexample systems and methods described herein, and/or equivalents. Theexample computing device may be a computer 500 that includes at leastone hardware processor 502, a memory 504, and input/output ports 510operably connected by a bus 508. In one example, the computer 500 mayinclude integration system logic 530 configured to facilitateintegrating ML models into a target application similar tosystem/methods 100 and 200 shown in FIGS. 1-4 .

In different examples, the logic 530 may be implemented in hardware, anon-transitory computer-readable medium 537 with stored instructions,firmware, and/or combinations thereof. While the logic 530 isillustrated as a hardware component attached to the bus 508, it is to beappreciated that in other embodiments, the logic 530 could beimplemented in the processor 502, stored in memory 504, or stored indisk 506.

In one embodiment, logic 530 or the computer is a means (e.g.,structure: hardware, non-transitory computer-readable medium, firmware)for performing the actions described. In some embodiments, the computingdevice may be a server operating in a cloud computing system, a serverconfigured in a Software as a Service (SaaS) architecture, a smartphone, laptop, tablet computing device, and so on.

The means may be implemented, for example, as an ASIC programmed tointegrate ML models into a target application similar to system/methods100 and 200 shown in FIGS. 1-4 . The means may also be implemented asstored computer executable instructions that are presented to computer500 as data 516 that are temporarily stored in memory 504 and thenexecuted by processor 502.

Logic 530 may also provide means (e.g., hardware, non-transitorycomputer-readable medium that stores executable instructions, firmware)for performing ML model integration.

Generally describing an example configuration of the computer 500, theprocessor 502 may be a variety of various processors including dualmicroprocessor and other multi-processor architectures. A memory 504 mayinclude volatile memory and/or non-volatile memory. Non-volatile memorymay include, for example, ROM, PROM, and so on. Volatile memory mayinclude, for example, RAM, SRAM, DRAM, and so on.

A storage disk 506 may be operably connected to the computer 500 via,for example, an input/output (I/O) interface (e.g., card, device) 518and an input/output port 510 that are controlled by at least aninput/output (I/O) controller 540. The disk 506 may be, for example, amagnetic disk drive, a solid state disk drive, a floppy disk drive, atape drive, a Zip drive, a flash memory card, a memory stick, and so on.Furthermore, the disk 506 may be a CD-ROM drive, a CD-R drive, a CD-RWdrive, a DVD ROM, and so on. The memory 504 can store a process 514and/or a data 516, for example. The disk 506 and/or the memory 504 canstore an operating system that controls and allocates resources of thecomputer 500.

The computer 500 may interact with, control, and/or be controlled byinput/output (I/O) devices via the input/output (I/O) controller 540,the I/O interfaces 518, and the input/output ports 510. Input/outputdevices may include, for example, one or more displays 570, printers 572(such as inkjet, laser, or 3D printers), audio output devices 574 (suchas speakers or headphones), text input devices 580 (such as keyboards),cursor control devices 582 for pointing and selection inputs (such asmice, trackballs, touch screens, joysticks, pointing sticks, electronicstyluses, electronic pen tablets), audio input devices 584 (such as

microphones or external audio players), video input devices 586 (such asvideo and still cameras, or external video players), image scanners 588,video cards (not shown), disks 506, network devices 520, and so on. Theinput/output ports 510 may include, for example, serial ports, parallelports, and USB ports.

The computer 500 can operate in a network environment and thus may beconnected to the network devices 520 via the I/O interfaces 518, and/orthe I/O ports 510. Through the network devices 520, the computer 500 mayinteract with a network 560. Through the network, the computer 500 maybe logically connected to remote computers 565. Networks with which thecomputer 500 may interact include, but are not limited to, a LAN, a WAN,and other networks.

Definitions and Other Embodiments

In another embodiment, the described methods and/or their equivalentsmay be implemented with computer executable instructions. Thus, in oneembodiment, a non-transitory computer readable/storage medium isconfigured with stored computer executable instructions of analgorithm/executable application that when executed by a machine(s)cause the machine(s) (and/or associated components) to perform themethod. Example machines include but are not limited to a processor, acomputer, a server operating in a cloud computing system, a serverconfigured in a Software as a Service (SaaS) architecture, a smartphone, and so on). In one embodiment, a computing device is implementedwith one or more executable algorithms that are configured to performany of the disclosed methods.

In one or more embodiments, the disclosed methods or their equivalentsare performed by either: computer hardware configured to perform themethod; or computer instructions embodied in a module stored in anon-transitory computer-readable medium where the instructions areconfigured as an executable algorithm configured to perform the methodwhen executed by at least a processor of a computing device.

While for purposes of simplicity of explanation, the illustratedmethodologies in the figures are shown and described as a series ofblocks of an algorithm, it is to be appreciated that the methodologiesare not limited by the order of the blocks. Some blocks can occur indifferent orders and/or concurrently with other blocks from that shownand described. Moreover, less than all the illustrated blocks may beused to implement an example methodology. Blocks may be combined orseparated into multiple actions/components. Furthermore, additionaland/or alternative methodologies can employ additional actions that arenot illustrated in blocks. The methods described herein are limited tostatutory subject matter under 35 U.S.C §101.

The following includes definitions of selected terms employed herein.The definitions include various examples and/or forms of components thatfall within the scope of a term and that may be used for implementation.The examples are not intended to be limiting. Both singular and pluralforms of terms may be within the definitions.

References to “one embodiment”, “an embodiment”, “one example”, “anexample”, and so on, indicate that the embodiment(s) or example(s) sodescribed may include a particular feature, structure, characteristic,property, element, or limitation, but that not every embodiment orexample necessarily includes that particular feature, structure,characteristic, property, element or limitation. Furthermore, repeateduse of the phrase “in one embodiment” does not necessarily refer to thesame embodiment, though it may.

A “data structure”, as used herein, is an organization of data in acomputing system that is stored in a memory, a storage device, or othercomputerized system. A data structure may be any one of, for example, adata field, a data file, a data array, a data record, a database, a datatable, a graph, a tree, a linked list, and so on. A data structure maybe formed from and contain many other data structures (e.g., a databaseincludes many data records). Other examples of data structures arepossible as well, in accordance with other embodiments.

“Computer-readable medium” or “computer storage medium”, as used herein,refers to a non-transitory medium that stores instructions and/or dataconfigured to perform one or more of the disclosed functions whenexecuted. Data may function as instructions in some embodiments. Acomputer-readable medium may take forms, including, but not limited to,non-volatile media, and volatile media. Non-volatile media may include,for example, optical disks, magnetic disks, and so on. Volatile mediamay include, for example, semiconductor memories, dynamic memory, and soon. Common forms of a computer-readable medium may include, but are notlimited to, a floppy disk, a flexible disk, a hard disk, a magnetictape, other magnetic medium, an application specific integrated circuit(ASIC), a programmable logic device, a compact disk (CD), other opticalmedium, a random access memory (RAM), a read only memory (ROM), a memorychip or card, a memory stick, solid state storage device (SSD), flashdrive, and other media from which a computer, a processor or otherelectronic device can function with. Each type of media, if selected forimplementation in one embodiment, may include stored instructions of analgorithm configured to perform one or more of the disclosed and/orclaimed functions. Computer-readable media described herein are limitedto statutory subject matter under 35 U.S.C §101.

“Logic”, as used herein, represents a component that is implemented withcomputer or electrical hardware, a non-transitory medium with storedinstructions of an executable application or program module, and/orcombinations of these to perform any of the functions or actions asdisclosed herein, and/or to cause a function or action from anotherlogic, method, and/or system to be performed as disclosed herein.Equivalent logic may include firmware, a microprocessor programmed withan algorithm, discrete logic (e.g., ASIC), at least one circuit, ananalog circuit, a digital circuit, a programmed logic device, a memorydevice containing instructions of an algorithm, and so on, any of whichmay be configured to perform one or more of the disclosed functions. Inone embodiment, logic may include one or more gates, combinations ofgates, or other circuit components configured to perform one or more ofthe disclosed functions. Where multiple logics are described, it may bepossible to incorporate the multiple logics into one logic. Similarly,where a single logic is described, it may be possible to distribute thatsingle logic between multiple logics. In one embodiment, one or more ofthese logics are corresponding structure associated with performing thedisclosed and/or claimed functions. Choice of which type of logic toimplement may be based on desired system conditions or specifications.For example, if greater speed is a consideration, then hardware would beselected to implement functions. If a lower cost is a consideration,then stored instructions/executable application would be selected toimplement the functions. Logic is limited to statutory subject matterunder 35 U.S.C. §101.

An “operable connection”, or a connection by which entities are“operably connected”, is one in which signals, physical communications,and/or logical communications may be sent and/or received. An operableconnection may include a physical interface, an electrical interface,and/or a data interface. An operable connection may include differingcombinations of interfaces and/or connections sufficient to allowoperable control. For example, two entities can be operably connected tocommunicate signals to each other directly or through one or moreintermediate entities (e.g., processor, operating system, logic,non-transitory computer-readable medium). Logical and/or physicalcommunication channels can be used to create an operable connection.

“User”, as used herein, includes but is not limited to one or morepersons, computers or other devices, or combinations of these.

While the disclosed embodiments have been illustrated and described inconsiderable detail, it is not the intention to restrict or in any waylimit the scope of the appended claims to such detail. It is, of course,not possible to describe every conceivable combination of components ormethodologies for purposes of describing the various aspects of thesubject matter. Therefore, the disclosure is not limited to the specificdetails or the illustrative examples shown and described. Thus, thisdisclosure is intended to embrace alterations, modifications, andvariations that fall within the scope of the appended claims, whichsatisfy the statutory subject matter requirements of 35 U.S.C. §101.

To the extent that the term “includes” or “including” is employed in thedetailed description or the claims, it is intended to be inclusive in amanner similar to the term “comprising” as that term is interpreted whenemployed as a transitional word in a claim.

To the extent that the term “or” is used in the detailed description orclaims (e.g., A or B) it is intended to mean “A or B or both”. When theapplicants intend to indicate “only A or B but not both” then the phrase“only A or B but not both” will be used. Thus, use of the term “or”herein is the inclusive, and not the exclusive use.

What is claimed is:
 1. A computing system, comprising: at least oneprocessor connected to at least one memory; a non-transitory computerreadable medium including instructions stored thereon that when executedby at least the processor cause the processor to: generate a userinterface configured to integrate a machine learning (ML) model into atarget application, wherein the user interface provides a list ofavailable ML models for selection on a display screen; in response to aselected ML model from the list of available ML models, retrieve adefinition file that represents the selected ML model; parse thedefinition file to (1) identify one or more input features defined forthe ML model, and (2) identify one or more output predictions that theML model is configured to generate; map the one or more input featuresof the ML model to one or more input data locations from the targetapplication, wherein the one more input data locations provide inputvalues to the one or more input features; map the one or more outputpredictions of the ML model to one or more result locations within thetarget application; generate an executable script configured as a datacommunication interface between the target application and the ML modeland to execute the ML model based on the mappings; wherein theexecutable script is configured to at least (i) retrieve input datavalues from the one or more input data locations from the targetapplication and pass the data values to the ML model as the one or moreinput features, and (ii) obtain the one or more output predictions fromthe ML model and insert the one or more output predictions into the oneor more result locations within the target application; and in responseto receiving the one or more output predictions, based on the executablescript, automatically insert the one or more output predictions in theone or more result locations within the target application.
 2. Thecomputing system of claim 1, wherein the definition file that representsthe selected ML model is a predictive model markup language (PMML) fileor an open neural network exchange (ONNX) file format.
 3. The computingsystem of claim 1, wherein the machine learning (ML) model is designatedto integrate into the target application on-demand; wherein the targetapplication is a multi-dimensional application that when executedon-demand generates one or more data sheets with rows and columns ofdata values, wherein each intersection of the rows and the columnsidentifies a cell location within a data sheet; wherein the executablescript is configured to retrieve the input data values from the one ormore input data locations that are cell locations within the one or moredata sheets; and wherein the executable script maps the input datavalues from the cell locations to the one or more input features of theML model.
 4. The computing system of claim 1, wherein the machinelearning (ML) model is integrated into the target application on-demand;and wherein the executable script is a groovy rule configured toretrieve the input data values from the one or more input data locationsthat correspond to one or more cell locations in one or more data formsdisplayed by the target application during execution of the targetapplication.
 5. The computing system of claim 1, wherein the executablescript is configured to execute on-demand; and wherein based on themappings, the computing system is configured to generate a secondexecutable script that is configured to execute as a batch process;wherein the second executable script is configured to retrieve the inputdata values from the one or more input data locations by querying adatabase associated with the target application and retrieving the inputdata values corresponding to the one or more input data locations fromthe database.
 6. The computing system of claim 1, wherein the targetapplication is a multi-dimensional planning application; and wherein theML model is configured to generate predictions based on data values fromthe multi-dimensional planning application that are mapped to the one ormore input features of the ML model.
 7. The computing system of claim 1,wherein executable script is configured to function as a data controllerand data communication interface to obtain data from selected datalocations and pass data between the target application and the ML model.8. A computer-implemented method performed by a computing deviceincluding at least one processor, the method comprising: generating, ona display device, a user interface configured to integrate a machinelearning (ML) model into a target application, wherein the userinterface provides a list of available ML models for selection on thedisplay device; in response to a selected ML model being selected fromthe list of available ML models, retrieving a definition file thatrepresents the selected ML model; parsing the definition file to (i)identify one or more input features defined for the ML model, and (ii)identify one or more output predictions that the ML model is configuredto generate; mapping, via the user interface, the one or more inputfeatures of the ML model to one or more input data locations from thetarget application, wherein the one more input data locations provideinput values to the one or more input features; mapping, via the userinterface, the one or more output predictions of the ML model to one ormore result locations associated with the target application; generatingand executing, by at least the computing device, an executable ruleincluding executable code that at least (i) defines the mappings betweenthe ML model and the target application, and (ii) transmits data valuesfrom the one or more input data locations to the ML model as input;executing, by at least the computing device, the ML model based on theinput to generate the one or more output predictions; and inserting theone or more output predictions into the one or more result locationsassociated with the target application.
 9. The method of claim 8,wherein generating the executable rule further includes: configuring theexecutable rule based on the mappings, wherein the executable rule isconfigured to include executable code that causes the computing deviceto perform functions comprising: (i) retrieving input data values fromthe one or more input data locations from the target application; (ii)transmitting the data values to the ML model as the one or more inputfeatures; (iii) obtaining the one or more output predictions from the MLmodel; and (iv) inserting the one or more output predictions into theone or more result locations within the target application.
 10. Themethod of claim 8, wherein the definition file that represents theselected ML model is a predictive model markup language (PMML) file oran open neural network exchange (ONNX) file format.
 11. The method ofclaim 8, wherein the user interface provides options for executing themachine learning (ML) model on-demand or as a batch process; wherein thetarget application is a multi-dimensional application that when executedon-demand generates one or more data sheets with rows and columns ofdata values, wherein each intersection of the rows and the columnsidentifies a cell location within a data sheet; in response to theon-demand option: configuring the executable rule to retrieve the inputdata values from the one or more input data locations that are celllocations within the one or more data sheets; and configuring theexecutable rule to transmit the input data values from the celllocations to the one or more input features of the ML model.
 12. Themethod of claim 8, wherein the user interface provides an option forexecuting the machine learning (ML) model on-demand or as a batchprocess; in response to the on-demand option: integrating the machinelearning (ML) model into the target application on-demand while thetarget application is executing; and configuring the executable rule toretrieve the input data values from the one or more input data locationsthat correspond to one or more cell locations in one or more data formsdisplayed by the target application during execution of the targetapplication.
 13. The method of claim 8, wherein the user interfaceprovides an option for executing the machine learning (ML) modelon-demand or as a batch process; in response to a selection of the batchprocess: configuring the executable rule to retrieve the input datavalues from the one or more input data locations by querying a databaseassociated with the target application and retrieving the input datavalues corresponding to the one or more input data locations from thedatabase; and wherein the executable rule and the ML model are executedwhile the target application is not executing.
 14. A non-transitorycomputer readable medium including instructions stored thereon that whenexecuted by at least the processor cause the processor to: generate auser interface configured to integrate a machine learning (ML) modelinto a target application; parse a definition file that represents theselected ML model to (1) identify one or more input features defined forthe ML model, and (2) identify one or more output predictions that theML model is configured to generate; map the one or more input featuresof the ML model to one or more input data locations from the targetapplication, wherein the one more input data locations provide inputvalues to the one or more input features; map the one or more outputpredictions of the ML model to one or more result locations within thetarget application; generate executable script configured to function asa data controller and data communication interface to obtain the inputvalues from the one or more input data locations and pass data betweenthe target application and the ML model; wherein the executable scriptis configured to execute the ML model with the input values and obtainthe one or more output predictions from the ML model; and in response toobtaining the one or more output predictions, based on the executablescript, insert the one or more output predictions in the one or moreresult locations within the target application.
 15. The non-transitorycomputer readable medium of claim 14, wherein the executable script isconfigured to at least (i) retrieve input data values from the one ormore input data locations from the target application and pass the datavalues to the ML model as the one or more input features, and (ii)obtain the one or more output predictions from the ML model andautomatically insert the one or more output predictions into the one ormore result locations within the target application.
 16. Thenon-transitory computer readable medium of claim 14, wherein thedefinition file that represents the selected ML model is a predictivemodel markup language (PMML) file or an open neural network exchange(ONNX) file format.
 17. The non-transitory computer readable medium ofclaim 14, wherein the machine learning (ML) model is designated tointegrate into the target application on-demand; wherein the targetapplication is a multi-dimensional application that when executedon-demand generates one or more data sheets with rows and columns ofdata values, wherein each intersection of the rows and the columnsidentifies a cell location within a data sheet; wherein the executablescript is configured to retrieve the input data values from the one ormore input data locations that are cell locations within the one or moredata sheets; and wherein the executable script maps the input datavalues from the cell locations to the one or more input features of theML model.
 18. The non-transitory computer readable medium of claim 14,wherein the machine learning (ML) model is integrated into the targetapplication on-demand; and wherein the executable script is a groovyrule configured to retrieve the input data values from the one or moreinput data locations that correspond to one or more cell locations inone or more data forms displayed by the target application duringexecution of the target application.
 19. The non-transitory computerreadable medium of claim 14, wherein the executable script is configuredto execute on-demand; and wherein based on the mappings, a secondexecutable script is generated that is configured to execute as a batchprocess; wherein the second executable script is configured to retrievethe input data values from the one or more input data locations byquerying a database associated with the target application andretrieving the input data values corresponding to the one or more inputdata locations from the database.
 20. The non-transitory computerreadable medium of claim 14, wherein the target application is amulti-dimensional planning application; and wherein the ML model isconfigured to generate predictions based on data values from themulti-dimensional planning application that are mapped to the one ormore input features of the ML model.